Robust Shift-and-Invert Preconditioning: Faster and More Sample Efficient Algorithms for Eigenvector Computation

In this paper we provide faster algorithms and improved sample complexities for approximating the top eigenvector of a matrix A>A. In particular we give the following results for computing an approximate eigenvector i.e. some x such that x>A>Ax ≥ (1− )λ1(AA): • Offline Eigenvector Estimation: Given an explicit matrix A ∈ Rn×d, we show how to compute an approximate top eigenvector in time Õ ([ nnz(A) + d sr(A) gap2 ] · log 1/ ) and Õ ([ nnz(A)(d sr(A)) √ gap ] · log 1/ ) . Here sr(A) is the stable rank, gap is the multiplicative gap between the largest and second largest eigenvalues, and Õ(·) hides log factors in d and gap. By separating the gap dependence from nnz(A) our first runtime improves classic iterative algorithms such as the power and Lanczos methods. It also improves on previous work separating the nnz(A) and gap terms using fast subspace embeddings [AC09, CW13] and stochastic optimization [Sha15c]. We obtain significantly improved dependencies on sr(A) and and our second running time improves this further when nnz(A) ≤ d sr(A) gap2 . • Online Eigenvector Estimation: Given a distribution D over vectors a ∈ R with covariance matrix Σ and a vector x0 which is an O(gap) approximate top eigenvector for Σ, we show how to compute an approximate eigenvector using Õ ( v(D) gap2 + v(D) gap· ) samples from D. Here v(D) is a natural notion of the variance of D. Combining our algorithm with a number of existing algorithms to initialize x0 we obtain improved sample complexity and runtime results under a variety of assumptions on D. Notably, we show that, for general distributions, our sample complexity result is asymptotically optimal we achieve optimal accuracy as a function of sample size as the number of samples grows large. We achieve our results using a general framework that we believe is of independent interest. We provide a robust analysis of the classic method of shift-and-invert preconditioning to reduce eigenvector computation to approximately solving a sequence of linear systems. We then apply variants of stochastic variance reduced gradient descent (SVRG) and additional recent advances in solving linear systems to achieve our claims. We believe our results suggest the generality and effectiveness of shift-and-invert based approaches and imply that further computational improvements may be reaped in practice. 1 ar X iv :1 51 0. 08 89 6v 1 [ cs .D S] 2 9 O ct 2 01 5